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Introduction 


The Trade Adjustment Act Community College and Career Training 
(TAACCCT) grant represented an unprecedented investment by the federal 
government in integrated postsecondary education and workforce training 
offered primarily by community and technical colleges. Between 2011 and 2018, 
256 grants totaling nearly $2 billion were awarded through four rounds of 
competitive grants. Ultimately, 630 community colleges were represented in the 
overall group of 729 colleges and universities funded by TAACCCT, with 
community colleges making up 85 percent of all postsecondary institutions 
securing these grants (Cohen, 2017). More than any time in their over 100 years 
of existence, the TAACCCT grant spotlighted the critical role of community 
colleges in responding to economic downturns and preparing workers for a future 
in which postsecondary education and credentials are a necessity. 


This brief presents results of a meta-analysis of quasi-experimental design (QED) 
evaluation studies to estimate the average effects of TAACCCT grants on four 
student outcomes: program completion, credential attainment, post-training 
employment, and pre- to post-training wage change. Complementing emerging 
evidence on TAACCCT reported by the Urban Institute (see Cohen et al., 2017; 
Durham et al., 2017; Eyster, Cohen, Mikelson, & Durham, 2017; and Durham, 
Eyster, Mikelson, & Cohen, 2017) and forthcoming results from ABT Associates’ 
national impact evaluation, we hope this brief contributes to a fuller 
understanding of the impact of TAACCCT on the outcomes of student 
participants, many of whom enrolled in community colleges to master skills 
needed to secure living-wage jobs in the aftermath of the Great Recession. 
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The TAACCCT Grant 


Seeking policy levers to help turn around the nation’s weakest economy since the 
Great Depression, Congress created the created TAACCCT and tasked the U.S. 
Department of Labor (DOL) to administer the grant program, in collaboration 
with the U.S. Department of Education (ED). Making an unprecedented 
investment in the nation’s community and technical colleges, the DOL sought to 
direct its federal-funding authority in the Trade Adjustment Act (TAA) to 
substantially increase college access and credential attainment for adult workers 
detrimentally impacted by the recession. Beginning October 1, 2011 and 
continuing through September 30, 2018, almost $2 billion was invested in 
integrating postsecondary education and workforce training into programs of 
study in a wide range of occupational fields, from manufacturing to healthcare to 
energy. The DOL charged grantees with implementing programs that would not 
only help students secure immediate employment but also create career 
pathways that could offer longer-term economic and other benefits to 
participants over a lifetime. Put succinctly, the major goal of TAACCCT was to 
“provide workers with the education and skills to succeed in high-wage, high- 
skill occupations” (DOL, 2016, p. 3). 


A specific charge of the TAACCCT grant was for community and technical 
colleges, and other postsecondary institutions, to enroll adult workers who had 
lost their jobs or who needed initial training or retraining to find employment in a 
dramatically changing workforce. From 2011 to 2018, four-year TAACCCT grants 
were made to all 50 states, Puerto Rico, and the District of Columbia (DC) with 
the stipulation that these federal funds be used to implement new or update 
programs of study using evidence-based innovations and strategies that increase 
the capacity of colleges to deliver more and better integrated postsecondary 
education and workforce training. Results reported by the DOL (2019) after the 
conclusion of the grant noted that TAACCCT supported an extensive amount of 
this activity. 


TAACCCT grants created nearly 2,700 new or redesigned programs and enrolled 
over 500,000 students who earned more than 350,000 credentials. Across all 
rounds, the DOL awarded $393,734,412 in institutional grants to 146 single 
institution grantees, $1,120,036,098 to 84 single-state consortia, and 
$412,014,608 to 26 multi-state consortia (DOL, 2016). Manufacturing and 
healthcare were the two sectors with the largest grant activity, followed in 
descending order by energy, information technology, transportation and 
logistics, green technology, and agriculture (DOL, 2016). 


Because TAACCCT sought to address the needs of individuals detrimentally 
impacted by the nation’s depressed economy, the student populations targeted 
for recruitment and enrollment reflected this phenomenon. The Solicitation for 
Grant Application (DOL, 2011) articulated that the TAACCCT should target 
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“workers who have lost their jobs or are threatened with job loss as a result of 
foreign trade” (p. 1). As the economy improved in an uneven way across the 
country, the TAACCCT grantees were encouraged to implement programs 
targeting individuals eligible for training under the “TAA for Workers” program, 
as well as veterans, students with disabilities, and students who qualified for Pell 
grants. However, throughout the entire grant period, the DOL maintained a 
consistent emphasis on enrolling non-traditional age students, including 
individuals who were unemployed or underemployed. Reporting participation 
for these diverse populations was also a priority of performance-reporting 
requirements, so student enrollment was reported by grantees by gender, race/ 
ethnicity, employment and enrollment status (full- or part-time), and veteran, 
disability, and Pell-eligibility status; however, evaluation results measuring the 
impact of the grants were not required for sub-populations, which is a point we 
will return to toward the conclusion of this paper. 


After three rounds of TAACCCT grant awards, Durham et al. (2017) reported the 
average age of TAACCCT participants was 31 years, and also described a higher 
proportion of men than women enrolled in grant-funded programs (60% 
compared to 40%, respectively). Whereas the distribution by race/ethnicity of 
TAACCCT participants was similar to the nation’s population as a whole, the 
proportion of grant participants identifying with a non-white racial or ethnic 
group was lower than the overall average enrollment of racially minoritized 
groups in community and technical colleges nationally (59% versus 49%, 
respectively). This statistic varied by round but never approached the level of 
racial/ethnic minority representation in student enrollment in two-year colleges 
nationwide. Given the preponderance of grant activity in manufacturing and 
other occupations historically predominated by males, it is not surprising that 
more total participants were male than female, although it seems TAACCCT 
could have done more to close this gender gap. Interestingly, TAACCCT grants 
tended to enroll slightly more full- than part-time students, which is counter to 
national enrollment statistics for community colleges (American Association of 
Community Colleges, 2019); however, this relationship flipped between round 
three and round four when a higher proportion of TAACCCT participants were 
part-time enrolled and also full-time employed (46%), mirroring national 
statistics and possibly also reflecting the improving national economy. 


Besides the occupational-technical programs created or improved with 
TAACCCT funding, grantees were encouraged to implement core elements 
identified as having sufficiently rigorous evidence to support federal funding. 
Shifting somewhat in foci across the rounds of the grant awards, some core 
elements consistently mentioned in the Solicitation for Grant Application (SGA) 
included the general categories of evidence-based designs; career pathways and 
stackable credentials; transfer and articulation; online and technology-enabled 
instruction; employer engagement; and strategic alignment with industry, 
governors, the public-workforce system, and others. These core elements help 
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organize and provide some consistency for the wide-ranging programmatic 
approaches that grantees chose to pursue in the TAACCCT grant. 
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Third-Party Evaluation of TAACCCT 


All rounds of the SGA asked applicants to implement and replicate evidence- 
based models, programs, and practices. The emphasis on evidence-based 
strategies seems to reflect the Obama administration’s “preference for 
competitive grants and evidence of effectiveness in designing its grant programs” 
(Haskins & Margolis, 2014, p. 192), mimicking the efforts by ED to apply the gold, 
silver, and bronze standard. However, this approach may not have accounted for 
the lack of evidence of the impact of a wide range of reforms in the context of the 
community college, including numrous reforms referenced in the TAACCCT 
SGA such as career pathways, Prior Learning Assessment (PLA, and intensive 
student services. Even so, a comprehensive approach to evaluation was seen as 
critical to advancing the TAACCCT grants, by providing funding for grantees to 
secure third-party evaluation that could measure implementation relative to 
student outcomes using designs that would produce evidence on what is 
working. 


Round 1 of TAACCCT asked applicants to use an evidence-based framework for 
preparing proposals, recognizing that levels of evidence can be strong, moderate 
and preliminary. These levels are aligned to the Clearinghouse for Labor 
Evaluation and Research (CLEAR) standards of evidence created by the United 
States Department of Labor (DOL). Supplementary guidance for blueprint for the 
TAACCCT evaluation advocated by ED was the i3 grant program that sought 
evidence of impact consistent with the levels of rigorous evidence used by the 
Institute for Education Sciences (IES). In this schema, strong evidence refers to 
research and evaluation that addresses causal inference and conclusions (i.e., 
high internal validity) with sufficient sites and participants to suggest that 
interventions can be scaled up. Well-implemented experiments and QEDs 
supporting the effectiveness of programs and reform strategies fit this definition. 
Moderate evidence is generated by experimental, QEDs, and correlational 
designs with strong statistical controls for selection bias that provide some 
information useful to causal inference and conclusions, but lack broader 
generalizability. The third level of evidence, preliminary, refers to research 
yielding promising evidence of limited generalizability that are based on 
descriptive tracking studies and pre- and post-treatment comparison studies 
(Zandniapour & Deterding, 2018). On their own, these studies lack sufficient 
quality evidence to support scaling up. 


The DOL also required grantees to provide annual performance report results for 
grant participants, as well as matched comparison groups. Rounds 2 through 4 
continued the focus on rigorous evidence begun in Round 1 but moved to a 
requirement for a third-party evaluation to estimate the impact of the grant on 
student outcomes. Applicants were required to submit an evaluation plan and 
budget focusing on both implementation and impact. DOL’s directive was to use 
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the most rigorous evaluation design feasible to estimate the effects of the grant, 
whenever possible using experimental or quasi-experimental design (QED). By 
QED, we mean study designs that are not experimental design in the form of 
randomized control trials but quasi-experimental in that use alternative designs 
that enable researchers to estimate causal effects when randomization is 
determined to be infeasible or inappropriate. The DOL also required national 
evaluation activities involving all TAACCCT grants to assess implementation 
and impact, with this aspect conducted by Urban Institute and ABT Associates. 
Review of the TAACCCT evaluation plans showed that most third-party 
evaluations intended to use propensity score matching (PSM), with 70% of 
Rounds 3 and 4 specifying PSM as the evaluation design most feasible to 
estimating causal impacts. Only about 17% of the evaluations planned to conduct 
correlational (non-causal) pre-post or outcomes-only studies, and even fewer 
included plans for experimental designs (Cohen et al., 2017). 


The TAACCCT grant program wrapped up one year prior to this writing but only 
a handful of TAACCCT evaluation studies having been published (see, for 
example, Bragg & Krismer, 2016). This study fills a critical void in understanding 
the overall impact of TAACCCT as a federal policy focusing extensively on 
community and technical colleges. 
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Purpose of the Study 


This brief describes findings from a meta-analysis of third-party evaluation (TPE) 
studies using QED designs to determine the impact of the TAACCCT grants on 
students’ educational and employment outcomes. The primary objective of this 
study was to examine the merits of a novel data source for meta-analysis—third- 
party evaluation reports using QED designs—on estimating the average effects of 
TAACCCT funding on student-level outcomes. Our second objective was to 
consider the implications of meta-analysis as a methodological approach for 
evaluating future postsecondary education and workforce training policy. We 
conclude with a brief discussion of the potential for improved alignment of large- 
scale federal investments with rigorous evaluation and meta-analytic designs 
that reveal effects, and we use this information for a second report offering 
recommendations for future federal investments in systematic and rigorous 
evaluation of community and technical college education (Bragg, 2019). 
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Meta-Analysis Methods 


Meta-analysis aggregates statistical findings across multiple studies. Used to 
inform such questions as “What does the research tell us about this 
intervention?” meta-analysis is commonly used to synthesize epidemiology and 
health research and increasingly found in the social sciences and education 
(Cooper, Hedges, & Valentine, 2009). A meta-analysis process distills hundreds 
of publications into a handful of studies that meet a set of criteria for inclusion. 
The preliminary data for this meta-analysis came from 216 TAACCCT evaluation 
reports (comprising 84% of all TAACCCT grants), from which 36 evaluation 
reports were selected for inclusion after a four-phase review process. The vast 
majority of final third-party evaluation reports were accessed from the 
SkillsCommons Repository (see:skillscommons.org), but a few reports were 
obtained directly from evaluators when a search of the SkillsCommons 
Repository did not yield a copy of the final report. The four-phase process for 
reviewing and making decisions on inclusion/exclusion is described below. 


In Phase One, the research team reviewed each publicly available TAACCCT 
evaluation report. Two reviewers read each report identifying the interventions, 
looking for evidence of a grant theory of change, quantitatively scoring 
implementation and outcomes, and identifying the evalaution design including 
comparison groups. 


In Phase Two, the reports that claimed to employ QED (n=143) were reviewed to 
determine whether they were plausibly QED studies. This phase required careful 
reading; many reports contained treatment and comparison groups because of 
DOL instructions that third-party evaluations should include comparison 
studies, but upon further inspection were determined by reviewers to not be 
QED. Recognizing that evaluators faced pressure to implement QEDs even in the 
face of data limitations and other methodological constraints, reviewers also 
detailed efforts to implement QEDs that were unsuccessful. An additional 
component of this phase was determining which outcomes might be sufficiently 
represented in the evaluation reports that a critical mass of data points were 
present. We determined the outcomes of program completion, credential 
completion, employment, and wage change to be the most meaningful and 
plentiful of measures used across the full spectrum of evaluation studies for 
TAACCCT, and we narrowed to these outcomes in the next phase of the review 
process. 


In Phase Three, the studies where authors claimed to have used any form of 
experimental and quasi-experimental design with causal estimates linking 
TAACCCT program participation to the outcomes of program completion, 
credential completion, employment, and wage change were evaluated for 
inclusion in our meta-analysis (n=66). The Phase Three review determined 
whether the studies included evidence of a QED, and in all cases but two were 


newamerica.org/education-policy/reports/estimating-impact-taaccct/ 12 


revealed to use propensity score matching (PSM). Given that the TAACCCT 
evaluation reports had not been published in refereed journals and therefore 
were not subject to the rigorous methodological review process associated with 
publication of meta-analysis studies in the literature, we established selection 
criteria that took into account factors normally considered in peer-refereed 
review. These criteria included: study authors identified their evaluation study 
design as a QED and described aspects of the QED design in the text of their 
evaluation report in sufficient depth and detail to provide reasonable confidence 
that the study design was actually a QED. Studies reviewed in Phase Three where 
the evaluation report lacked sufficient statistical information consistent with the 
purported approach to QED were eliminated from review. However, a sub-set of 
QED studies that lacked a specific statistic needed for the meta-analysis (i.e., 
lacking a sample size or standard error statistic) but otherwise which appeared to 
be a bona fide QED design, were identified as a potential QED and the authors of 
these studies, totaling 15, were contacted via email and phone to request the 
missing statistic that would allow us to include the study. All but four authors 
provided the requested statistic so that these studies could be included in our 
meta-analysis. This step was important, as these added studies made up 31% of 
the total evaluations included here. 


Figure 1. The Four Phases of Meta-Analysis Process 


Evaluation reports in Skills Commons Evaluation reports obtained from authors 
(n=208) (n=8) 


aa ot 


Total evaluation reports -+—>|_ Evaluation reports excluded due to 
screened (n=216) descriptive analysis only or not the 


ee final report (n=73) 


Evaluation reports assessed >| Evaluation reports excluded due to 
for eligibility (n=143) not QED or no evidence of QED 


a =n 


Evaluation reports considered [——>} 
for meta-analysis (n=66) 


Evaluation reports excluded (n=30) 
for following reasons: 
Not QED or no evidence of QED 
(n=15); 
Needed stats not reported or unclear 
Evaluation reports included and unable to obtain from author 
in quantitative synthesis (n=7); 
(meta-analysis) (n=36) DiD unableto include (n=3); & 
Non-binary outcome (n=1) 


Phase Four entailed conducting the meta-analysis using the 36 studies that met 
our inclusion criteria. At this point, each report’s outcomes (relative to treatment 
and comparison groups) were recorded and converted to standardized effects. 
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Studies that were not included in the final meta-analysis either lacked requisite 
statistics for inclusion or, after all evidence was examined, were determined to be 
descriptive in nature and actually not QED studies. Ultimately, 60 effects 
representing education and employment outcomes were drawn from the final set 
of 36 studies included in the meta-analysis. 


Table 1 provides a summary of the TAACCCT grants included in the meta- 
analysis study by grant title, author(s), round, grant type (single institution, state 
consortium or multi-state consortium), industry sector, and outcome(s). No QED 
studies were included from Round 1, but numerous studies were included from 
each of the other three rounds, with 14 studies from Round 2, 9 studies from 
Round 3, and 13 studies from Round 4. The breakdown of grant type included 12 
studies conducted of TAACCCT grants awarded to a single institution, 15 studies 
of single-state consortia, and 9 studies of multi-state consortia. As with the 
overall TAACCCT grants, manufacturing was the most prevalent industry sector 
included in the evaluations, with 8 studies focusing on healthcare, 5 on 
information technology (IT), and 4 multi-sector grants that also included 
manufacturing, healthcare, and IT. Other sectors represented in the grants 
included energy; transportation, distribution and logistics; business services, 
various trades (i.e., construction, electronics), and various other occupational 
areas. 


Table 1. Summary of TAACCCT Grant Characteristics in Meta-Analysis Study 


TAACCCT Grant Industry 
Round Title Author(s) Grant Type Location Sector Outcomes 


PTB & AL&FL —e Welding SC reventel 
Multi-state 


leti 
2 AF-TEN Associates Caneanie (5 e Industrial Bee es 
(2016) Colleges) Electronics 
Employment 


e Program 
Completion 
2 Allee neath Caffey (2016) single : NM Healthcare : 
Expansion Institution Employment 
e Wage 
Change 


e Program 
Completion 
AME State MN (3 e 


2 é Ho (2016 : Manufacturi 
Manufacturing ( ) Consortium Colleges) ee ca Employment 
e Wage 


Change 
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TAACCCT Grant 


Round Title 
Amplifying 

2 
Montana 
Competency- 

9 Based Education 
in Community 
Colleges 
Consortium for 

2 : : 

Bioscience 

a CT Health and Life 
Sciences 

7 lowa Advanced 
Manufacturing 

2 Making the Future 

. Minneapolis 
MAAC 

2 Online2Workforce 


Author(s) 


Feldman, 
Staklis, Hong, 
& Elrahman 
(2016) 


Person, 
Thomes, 
Bruch, 
Johann, & 
Maestas 
(2016) 


Alamprese, 
Costelloe, 
Price, & 
Zeidenberg 
(2017) 


Mokher & 
Pearson 
(2016) 


Mora, Kemis, 
Callen, & 
Starobin 
(2016) 


Price, Sedlak, 
Roberts & 
Childress 
(2016) 


Kundin & 
Dretzke 
(2016) 


Jensen, 
Horohov, & 
Wright (2016) 


Grant Type 


Single 


Institution 


Multi-state 
Consortium 


Multi-state 
Consortium 


State 
Consortium 


State 
Consortium 


State 
Consortium 


Single 
Institution 


Single 
Institution 


Location 


MT 


FL, OH, 
& TX (3 
Colleges) 


CA, FL, 
IN, NC, 
PA, TX, 
UT, WI 
(12 
Colleges) 


Crs 


Colleges) 


IA (15 
Colleges) 


WI (16 
Colleges) 


MN 


KY 
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Industry 
Sector 


Manufacturing 


Information 
Technology 
(IT) 


Bioscience 


Health 
Sciences 


Manufacturing 


Manufacturing 


Manufacturing 


Business 
Services 


Outcomes 


Program 
Completion 


Credential 
Completion 


Credential 
Completion 
(2 effects) 


Credential 
Completion 


Credential 
Completion 


e Credential 
Completion 
e 


Employment 


Program 
Completion 
(2 effects) 


e Credential 
Attainment 
e 


Employment 


15 


Round 


TAACCCT Grant 
Title 


Project IMPACT 


ShaleNET 


Retraining the 
Gulf Workforce 


Bridging the Gap 


Florida XCEL-IT 


Golden Triangle 


HOPE 


LA Healthcare 


Maine is IT 


Mission Critical 
Operations 


Author(s) 


Shain & 
Grandgenett 
(2016) 


Dunham et al. 


(2016) 


Patnaik & 
Prince (2016) 


Bellville et al. 
(2017) 


Swan et al. 
(2017) 


Harpole 
(2017) 


Good & Yeh- 
Ho (2017) 


Tan & Moore 
(2017) 


Horwood, 
Usher, 
McKinney, & 
Passa (2017) 


NC State 
Industry 
Expansion 
Solutions 
(2017) 


Grant Type 


State 
Consortium 


Multi-State 
Consortium 


Multi-State 
Consortium 


State 
Consortium 


State 
consortium 


Single 
Institution 


Multi-state 
Consortium 


State 
Consortium 


State 
Consortium 


Multi-State 
Consortium 


Location 


NE (5 
Colleges) 


PA (4 
Colleges) 


LA &MS 


WV (9 
Colleges) 


FL (7 
Colleges) 


MS 


FL, MN & 
MI (5 
Colleges) 


CA (8 
Colleges) 
ME (7 


Colleges) 


NC &GA 
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Industry 
Sector 


Manufacturing 


Energy 


Information 
Technology 
(IT) 


Multi-Sector 


Information 
Technology 
(IT) 


Manufacturing 


Healthcare 


Healthcare 


Information 
Technology 
(IT) 


e Information 
Technology 
e Engineering 


Outcomes 


Credential 
Completion 


Employment 


Credential 
Completion 


Credential 
Completion 


e Program 
Completion 
e Wage 
Change 


e Program 
Completion 


Employment 
Program 
Completion 
(5 effects) 
Program 


Completion 


Program 
Completion 


Program 
Completion 


16 


Round 


TAACCCT Grant 
Title 


North Dakota AM 


Northeast 
Resiliency 
Consortium 


Advancing Career 
& Training (ACT) 
for Healthcare 


Adult 
Competency- 
Based Education 
(ACED) 


Advanced 
Manufacturing for 
Global Economy 


Building Illinois 
Bioeconomy 


Greater Memphis 
Alliance 


Healthcare 
Careers Work 


Author(s) 


Jensen, 
Horohov, & 
Wright (2016) 


Price, 
Childress, 
Sedlak, & 
Roach (2017) 


Price, 
Valentine, 
Sedlak, & 
Roberts 
(2018) 


Bragg, 
Cosgrove, 
Cosgrove, & 
Blume (2018) 


Haviland, Van 
Noy, Kuang, 
Vinton, & 
Pardalis 
(2018) 


New Growth 
Group (2018) 


Patnaik ( 
2018) - 
impact; 
Juniper, C. 
(2018) 


WorkEd 
Consulting 
(2018) 


Grant Type 
Single 


institution 


Multi-state 
Consortium 


State 
Consortium 


Single 
Institution 


Single 
Institution 


State 
Consortium 


Multi-state 
Consortium 


Single 
Institution 


Location 


KY 


Nu (7 
Colleges) 


WI (16 
Colleges) 


UT 


OH 


IL (5 
Colleges) 


AR & TN 
(4 
Colleges) 


GA 
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Industry 
Sector 


Manufacturing 


Multi-Sector 


Healthcare 


Multi-Sector 


Manufacturing 


Resource 
Management 


e 
Manufacturing 
e Transport, 
Distribution & 
Logistics 


Healthcare 


Outcomes 


Program 
Completion 


e Program 
Completion 
e Credential 
Completion 
(2 effects) 


e Credential 
Completion 
e 
Employment 
e Wage 
Change 


e Program 
Completion 
e 


Employment 


Program 
Completion 


Program 
Completion 


Credential 
Completion 


Program 
Completion 
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TAACCCT Grant 


Round Title 

4 Heroes for Hire 

4 | Am STAR 

4 Kan-TRAIN 

4 New Mexico SUN 
4 Ohio Tech Net 


Plugged-In and 
Ready to Work 


UDC Construction 
and Hospitality 


Author(s) 


Horwood, 
Campbell, 
McKinney, & 
Bishop (2018) 


Dockery et al. 


(2018) 


Foster, 
Staklis, Ott & 
Moyer (2018) 


Dauphinee et 
al. (2018) 


The New 
Growth 
Group & The 
Ohio 
Education 
Resource 
Center (2018) 


Styers, 
Haden, 
Cosby, & 
Peery. (2018) 


Hendricks, 
Mitran, & 
Ferroggiaro 
(2018) 


Grant Type 


State 
Consortium 


Single 
institution 


State 


Consortium 


State 
Consortium 


State 
Consortium 


Single 
Institution 


Single 
Institution 


Location 


WV (3 
Colleges) 


OH 


KS (5 
Colleges) 


NM (11 
Colleges) 


OH (11 
Colleges) 


VA 


DC 
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Industry 
Sector 


e Healthcare 
e 


Manufacturing 


Manufacturing 


Multi-Sector 


Healthcare 


Manufacturing 


Manufacturing 


e 
Construction 
e Hospitality 


Outcomes 


Program 
Completion 


Employment 


e Credential 
Completion 
e 


Employment 


e Program 
Completion 
e 
Employment 
e Wage 
Change 


Program 
Completion 


Employment 


e Program 

Completion 
e Credential 
Completion 
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The relevant outcomes from qualified third-party evaluation reports were 
converted to a standardized effect in the form of an odds ratio and its associated 
standard error for the four outcomes: program completion, credential 
completion, post-program employment, and pre- to post-program wage change. 
Program completion and credential completion effects were pooled under the 
broader category of educational outcomes. Post-program employment and post- 
program wage change were likewise pooled and categorized as employment 
outcomes. 


Using a random effects model, these standardized effects for education and 
employment outcomes were weighted and pooled, resulting in “forest plot” 
visualizations and an estimate of the overall effect for the two outcomes of 
interest. In addition, a measure of heterogeneity was calculated to gauge the 
consistency of the studies’ results across the education and employment 
outcomes analyzed in this brief. Appendix A provides additional technical 
narrative to supplement the description of our approach for this study. Even 
though more options for conducting meta-analysis are rapidly emerging as this 
form of research in education grows (including the emergence of various forms 
meta-regression), we chose a fairly straightforward approach to this initial meta- 
analysis and may advance to more advanced forms assuming the data support 
these analytical designs. 
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Findings 


A series of forest plots represent our meta-analysis results for the selected 
TAACCCT grant evaluation studies. Figure 1 portrays educational outcomes, 
comprising program completion and credential completion, across 32 evaluation 
reports containing 41 effect sizes. The number “treated” by TAACCCT 
participation total 38,589 observations. Figure 2 illustrates an overall positive 
effect of TAACCCT participation on education outcomes (ESOR = 1.905; z = 5.18, 
p<0.001). 


Figure 2. Meta-analysis of Education Outcomes 


Study % 
ID ES (95% Cl) Weight 
Credential Completion 

‘Cons ortium for BioScience B 0.42 (0.29, 0.91) 243 
‘Consortium for BioScience A 0.55 (0.37, 0.81) 262 
Competency-Based Education in Community Colleges 0.22(0.75,089 280 
(CT Health and Life Sciences 1.14 (0.91, 1.42) 274 
Northeast Resiliency Consortium B 1.18 (1.02, 1.35) 278 
Kan-TRAIN 1.56 (1.33, 1.81) 278 
Retraining the Gulf Workforce 1.53 (1.40,181) 279 
Grester Memphis Alliance 1.78 (1.43,215) 275 
Wisconsin Making the Future A 2.15 (1.94, 239) 279 
UDC Construction and Hos pitslity 2.22(1.82,324 263 
Online2W orkforce 2.30 (1.66, 3.18) 267 
lows Advanced Msnufscturing 2.22(1.24,435 237 
ACT for Healthcare A 2.74 (2.42, 3.10) 273 
Bridging the Gap 2.81 (1.71,462) 252 
Northeast Resiliency Consortium A 485 (4.12,571) 277 
Project IMPACT 11.75 (5.71,2418) 225 
Program Completion 


‘Adult Competency Based Education A 
Minneapolis MAAC B 
Building Illinois Bioeconomy 


0.74 (0.60,0.91) 275 
0.75 (0.41, 1.38) 239 
0.85 (0.40, 1.78) 222 


Amplifying Montons 0.90 (0.58, 1.38) 258 
Northeast Resiliency Consortium 0.95 (0.81,1.12) 277 
UDC Construction and Hospitality 0.98 (0.69, 1.40) 265 
Mis sion Critical Operations 1.10 (0.96, 1.25) 278 
HOPE, College D 1.19(0.37,383 91.71 
AME Manufacturing A 1.37 (0.90,207) 260 
HOPE, College A 1.40 (0.13, 14.74) 0.78 
Maine is IT 1.45 (0.76,276) 235 
Ohio Tech Net 1.66 (1.37, 198) 276 


Healthcare Careers Work 
Allied Health Expansion A 


1.72 (1.31, 226) 271 
1.77 (0.99, 3.17) 242 


Heroes for Hire 1.82 (1.26, 262) 264 
HOPE, College B 1.83 (0.63, 520) 1.83 
North Dakota AM 2.30 (1.40, 379) 251 
HOPE, College C 2.88 (0.77,10.77) 1.54 
Minnespolis MAAC A 3.76 (1.47, 962) 1.98 
New Mexico SUN 4.58 (4.05, 519) 279 
Adv. Manufacturing for Global Economy 6.27 (2.25, 17.46) 188 
LA Healthcare 6.67 (6.08, 7.35) 279 
Hope, College E 6.91 (0.83, 57.34) 0.90 
Florida XCEL-IT A 7.67 (4.75, 12.40) 253 
Golden Triangle 1087 (7.30, 1558) 263 


Overall (I-squared = 97.9%, p= 0.000) 1.91 (1.49, 243) 100.00 


NOTE: Weights are from random effects analysis 


Figure 3 portrays a similar, positive effect of TAACCCT participation relative to 
the odds of employment outcomes. Of the 15 evaluation reports from which 18 
effects were identified and included in the meta-analysis, TAACCCT 
participation was associated with statistically significant and positive increase in 
the odds of employment outcomes (ESOR = 1.273; z = 2.00, p = 0.046). These 18 
employment-related effects comprised 8,870 TAACCCT observations. 
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Figure 3. Meta-analysis of Employment Outcomes 


Study % 
ID ES (95% Cl) | Weight 
Employment ‘ 
Adult Competency Based Education B —— H 0.57 (0.46, 0.70) 6.88 
Online2W orkforce ——_ i 0.61 (0.38, 0.98) 5.64 
AME Manufacturing B Oo 0.86 (0.47, 1.58) 4.91 
Project IMPACT (ee 0.86 (0.27, 2.80) 2.60 
Kan-TRAIN + | 0.89 (0.69, 1.14) 6.73 
Allied Health Expansion B p<. 1.11 0.51, 2.38) 4.12 
Wisconsin Making the Future B oe 1.21 (0.97, 1.51) 6.84 
ACT for Healthcare B — 1.40 (1.08, 1.82) 6.67 
1 Am STAR —+ 1.54 (0.47, 5.01) 2.58 
ShaleNET a 1.81 (1.13, 2.91) 5.64 
Pluggedin and Ready to Work ————— 2.12 (0.96, 4.70) 3.98 
New Mexico SUN ' ao 2.19 (1.96, 246) 7.15 
Golden Triangle —— 3.68 (2.67, 5.08) 6.41 
. i] 
Wage Gain 
Florida XCELIT —_ a ! 0.83 (0.57, 1.21) 6.14 
ACT for Healthcare C = 1.29 (1.07, 1.56) 6.95 
New Mexico Sun = 1.30 (1.17, 1.46) 7.15 
Allied Health Expansion C + 1.64 (1.06, 253) 5.84 
AME Manufacturing C ==> = 1.64 (0.71, 3.81) 3.78 
i] 
Overall (I-squared = 91.9%, p = 0.000) kKh> 1.27 (1.00, 1.61) 100.00 
‘ 
NOTE: Weights are from random = analysis } 
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Funnel plots were used to test for biased results for educational and employment 
outcomes separately (Figure 4). In the traditional sense publication bias can be 
considered moot here because our meta-analysis is comprised of evaluation 
reports not published in peer-refereed journals. However, publication bias could 
be relevant nonetheless because of the pressure evaluators face to report positive, 
statistically significant findings (Scriven, 2001). In other words, TAACCCT 
evaluators could have included only positive significant results of program 
participation and suppressed or omitted negative findings. An asymmetric 
distribution of effects in a meta-analysis funnel plot suggests publication bias; the 
funnel plots generated for the education outcomes (Figure 4) and employment 
outcomes (Figure 5) suggest no visual asymmetry (1.e., bias) in how effects in the 
evaluation reports. The commonly used Egger test (Egger et al., 1997) provides 
evidence of this lack of bias for both education (-0.381, ¢ = -0.21, p = 0.832) and 
employment (-1.140, t = -0.80, p = 0.438) outcomes. 
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Figure 4. Funnel Plot of Education Outcome Effects 
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Figure 5. Funnel Plot of Employment Outcome Effects 
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In addition to examining the possibility of biased results in the TAACCCT 
evaluation reports included in this meta-analysis, an I? statistic was calculated to 
gauge heterogeneity across the education and employment effects. The I? 
statistic provides a "measure of inconsistency in the studies’ results" and 
captures the percentage of total variation across studies due to heterogeneity 
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rather than chance (Higgins & Thompson, 2002). TAACCCT evaluation reports 
of varying sample sizes, study designs, and outcome measures means the I- 
squared measure is well suited to gauge heterogeneity since the measure was 
developed for these common characteristics (Higgins et al. 2003). 
Heterogeneous effects appear to be a characteristic of the education (97.9%, p < 
0.001) and employment (91.9%, p < 0.001) outcomes. 


The Appendix provides a summary of results of all studies by grant title, 
author(s), comparison sample, total analytic sample size, and effects size and also 
standard error by outcome for each study. This summary table provides a concise 
overview of the educational outcomes in the meta-analysis, including program 
completion and credential completion, and employment outcomes meta- 
analysis, including employment and wage change. 


The final analysis conducted as part of this meta-analysis was a content analysis 
of the core elements purported by third-party evaluators to be implemented in 
the TAACCCT grants. This analysis involved two members of the research team 
independently reading and coding the grants using a rubric that identified SGA- 
specified core elements of all four rounds of the TAACCCT grants. Table 2 
shows these results in the form of a “heat map,” which is a common way of 
visually displaying results associated with meta-analysis. Heat maps supplement 
meta-analysis studies by presenting the incidence of implementation of program 
elements, as used in our approach. In our case, the evaluation studies refer to 
core elements as part of a theory of change as well as planned implementation 
and provide varied levels of detail as to the extent to which they were actually 
implemented. Therefore, these heat maps should be interpreted as providing 
insights into core elements that may have some level of relationship to the 
average effects of the meta-analysis studies but not providing irrefutable 
evidence of impact. 


The heat maps associated with this study show the prevalence of core elements 
implemented in TAACCCT grants. Occurring most frequently in these grants is 
core elements associated with career pathways, specifically stacked and latticed 
credentials, prior learning assessment (PLA), comprehensive student supports, 
and career and employment services and supports. Several core elements in the 
grant are associated with online learning, assessment, and open educational 
resources (OER), online and technology-enabled learning and, to a lesser extent, 
accelerated online learning. Also prevalent in the grants is employer engagement 
and workforce agency partnerships. Sustainability and evaluation are also 
documented in the grants. Taken together, these core elements give a sense of 
the grant-funded core elements and strategies that were implemented in 
programs funded by TAACCCT. They are not presumed to cause the positive 
outcomes but they are potential contributors to impact, and they deserve 
consideration in future evidence-based federal investments such as TAACCCT. 
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Heat map 
Mo% 125% 26-50% (51-75% i 76-100% 


Credential 
Program Completion Completion Employment Wage Change 
Core Elements 19 Studies 17 Studies 16 Studies 5 Studies 


Career Pathways 


Stacked & latticed Credentials 


Integrated Basic Skills & Career Training 
Development Ed reform 

Non-credit to credit bridges 

CBE, CB assessments & CB training courses 
Modularlized curriculum 

Credit for Prior Learning & PLA 
Comprehensive student success 

Career guidance & services 


Pre-apprenticeships & Apprenticeships 


Work-based learning 


Transfer & Articulation 


Transfer and articulation of credit 


Applied baccalaureate degrees 


Retention Strategies 


Accelerated program model 


Self-paced learning 
Restructured course scheduling 


Online & OER 


Online CBE & CB assessments 

Rolling & open enrollment 

Accelerated course delivery/Hybrid 
Asynchronous & Real-time collaborations 
Online & technology-enabled learning 
Educational gaming 

Tech-enabled tutoring 

MOOCs 

Personalized tech-enabled instruction 
"Next Gen" assessment 
Technology-based infrastructure projects 


OER 


Partnerships 
Employer engagement 
Explicit use of sector strategies 

Public workforce partnering 

State gov/Governor partnering 
Philanthropic/External funding partnering 
Labor partnering 


TAACCCT grant partnering 


Evaluation & Sustainability 


Evaluation & Continuous Improvement 


Sustainability 


NEW AMERICA 
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Discussion 


These results suggest TAACCCT had a positive effect on both the education and 
employment outcomes of those who participated in its funded programs and 
strategies. The magnitude of TAACCCT’s positive effect appears greater for the 
educational outcomes of program completion and credential completion 
compared to the employment outcomes, measured as post-program employment 
and pre- to post-program wage change. These findings, resulting from a thorough 
review of more than 200 TAACCCT evaluation reports, make an important 
contribution to estimating the impact of the TAACCCT grant’s federal 
investment in community and technical colleges. 


Although positive results emerged from the meta-analysis, it is important to heed 
the caution of Pigott (2012, p. 146), a noted expert in meta-analysis techniques, 
who observes that “even if a systematic review yields equivocal results, a well- 
conducted systematic review should help to illuminate the issues” related to the 
effects. This study is limited to the subset of QED studies that provided sufficient 
methodological and statistical information for inclusion in the analysis and 
therefore should not be assumed to necessarily represent all evaluation results. 
Thus, these results should not be expected to stand alone as a single source 
guiding decision-making about future policy and program design but 
complement the national evaluation efforts of the Urban Institute and ABT 
research teams who are continue to conduct independent evaluation studies to 
describe implementation and measure the impact of TAACCCT. 


Complementing the quantitative results of the meta-analysis is a heat map, 
providing insight into the core elements implemented in the TAACCCT grants. 
In this analysis, the career pathways approach was implemented widely, 
including stackable and latticed credentials, prior learning assessment (PLA), 
comprehensive student supports, career advising and guidance, and employment 
supports. Online and technology-enabled learning and related online strategies 
were also implemented in a substantial number of the studies included in the 
meta-analysis. Partnerships between the community and technical colleges and 
employers and workforce agencies were also mentioned, and strategies 
emphasizing the importance of sustaining elements of the grant into the future 
was evident in the evaluation reports. 


Evaluation was also mentioned as an important strategy associated with 
TAACCCT, with numerous grant reports offering recommendations to help 
grantees continue to track the educational and employment outcomes of grant- 
funded participants and add new participants into the tracking of these student 
outcomes in the future. Evaluators mentioned in their reports the importance of 
continuing to assess student outcomes, often recognizing complexities in 
carrying out rigorous impact evaluation. 
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Rigorous third-party evaluations using QED are challenging to conduct. As many 
as 70% of third-party evaluators made plans to implement a rigorous design for 
grants in Rounds 3 and 4, but many fewer were executed due to methodological, 
logistical, and other difficulties. Barriers that impeded the execution of rigorous 
evaluation designs are more fully explicated in Bragg (2019) in a companion brief 
that offers recommendations for future federal investments in community and 
technical colleges. These suggestions include a call for the federal government to 
establish realistic expectations and fund professional development and technical 
assistance that helps evaluators execute their studies usingdesigns that 
illuminate a program’s impact on targeted student populations, both in aggregate 
and broken down for sub-populations so that a determination can be made about 
whether outcomes are distributed equitably. More nuanced designs that delve 
into the impact of core elements, such as PLA and comprehensive student 
supports are also needed, and this idea is supported by the two briefs completed 
by members of the New America team (Palmer, Nguyen 2019; Love 2019). 
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Appendix: Technical Notation 


The preponderance of meta-analysis studies to synthesize research findings on 
education and workforce interventions have generally focused on student-level 
interventions, not public policy and its effects on student outcomes (Tight, 2018). 
The United States Department of Labor’s (DOL) Trade Adjustment Assistance 
Community College and Career Training (TAACCCT) grants to community 
colleges, plus the DOL’s evaluation requirement of TAACCCT grants, thus 
presents a unique opportunity to aggregate empirical findings across grantees to 
isolate the grant program’s broader effects on key policy-relevant outcomes. 


For each effect included in the meta-analysis, a conventional log odds ratio was 
calculated as: 


where for effect i from a given evaluation report, a and b are the participant 
counts of those in the treatment and comparison groups, respectively, achieving 
the outcome of interest (program completion, post-program employment, etc.); ¢ 
and d are the participant counts for the treatment and comparison groups not 
achieving the outcome of interest. Using these counts for effect i, the standard 
error for each log odds ratio can also be calculated as (Pigott, 2012): 


se{In(OR;)} = /1/a\+ 1/b; + 1/¢, + 1/d, 


These odds ratios and standard errors were correspondingly analyzed using a 
random effects meta-analysis model (following DerSimonian & Laird, 1986). A 
random effects model was chosen, assuming that this analysis would be 
capturing a distribution of effects from TAACCCT, not one common or “true” 
effect (Borenstein et al., 2010). 


To implement a random effects meta-analysis, a variance 1 ? for a distribution of 
observed treatment effects, assumed 9 i~ N( 8, 1 2), was considered relative to 
calculating a weight for each study’s effect size: 


w; = 1/{se(6,) + #7} 
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These weights, in turn, allow for an overall effect size and standard error given 
as: 


6pop = (Yw,8;)/(Xwi) 
se{6pop} = 1//Lw; 


These overall effect sizes and standard errors yield the estimates that are used to 
generalize about a potential effect of TAACCCT funding on education and 
employment outcomes of interest. 


In addition to the meta-analysis overall effect, an I? test statistic was calculated 
to provide a "measure of inconsistency in the studies’ results" and capture the 
percentage of total variation across studies that is due to heterogeneity rather 
than chance (Higgins & Thompson 2002). The meta-analysis data collected from 
TAACCCT evaluation reports contained varying sample sizes, study designs, and 
outcome measures; the I? statistic is well suited to gauge possible heterogeneity 
to arise from these factors since the measure was developed for these common 
characteristics of meta-analysis (Higgins et al. 2003). This heterogeneity measure 
is calculated as: 


I? = 100% x (Q—df)/Q 


where Q is the conventional Cochran's heterogeneity statistic and dfis degrees of 
freedom, i.e., number of effects in the given meta-analysis. Cochran’s 
heterogeneity statistic is calculated by taking the sum of the squared deviations 
of each study's estimate from the meta-analytic overall effect and weighting each 
study's contribution in a manner that mirrors the meta-analysis weights 
(Cochran, 1954). Negative values for I2 are set to zero such that the measure 
ranges from 0 to 100%. 


Higgins et al. (2002) note that heterogeneity above 75% is considered high and 
that only a quarter of all published meta-analysis have I2 values above 50%. A low 
level of heterogeneity shows an overall effect, reflecting results from studies that 
are largely homogeneous in terms of their effects’ direction, magnitude, and 
significance. A high level of heterogeneity, on the other hand, means that an 
overall effect may be positive and significant but results from highly inconsistent 
effects. Higgins et al. (2003) note that generalizing from meta-analysis largely 
depends on overall effects that are the result of consistent results. 
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This brief’s education and workforce meta-analyses exhibit highly 
heterogeneous overall effects. Such heterogeneity is intuitive, given the diverse 
effects portrayed in forest plots. The education outcome’s heterogeneity (98.0%, 
p<o.001), for instance, contains 20 effects that are positive and statistically 
significant compared to 18 effects that are statistically insignificant (7 negative 
and 11 positive). This yields an overall effect that is positive, statistically 
significant, and highly heterogeneous. The effect of TAACCCT on employment 
likewise has a similar I2 (92.4%) that reflects an overall effect that is positive and 
statistically significant (p<o.001). 
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Notes 
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